Kalman filter control in the reinforcement learning framework

There is a growing interest in using Kalman-filter models in brain modelling. In turn, it is of considerable importance to make Kalman-filters amenable for reinforcement learning. In the usual formulation of optimal control it is computed off-line by solving a backward recursion. In this technical note we show that slight modification of the linear-quadratic-Gaussian Kalman-filter model allows the on-line estimation of optimal control and makes the bridge to reinforcement learning. Moreover, the learning rule for value estimation assumes a Hebbian form weighted by the error of the value estimation. 1. Motivation Kalman filters and their various extensions are well studied and widely applied tools in both state estimation and control. Recently, there is an increasing interest in Kalman-filters or Kalman-filter like structures as models for neurobiological substrates. It has been suggested that Kalman-filtering (i) may occur at sensory processing [6, 7], (ii) may be the underlying computation of the hippocampus, and may be the underlying principle in control architectures [8, 9]. Detailed architectural similarities between Kalman-filter and the entorhinal-hippocampal loop as well as between Kalman-filters and the neocortical hierarchy have been described recently [2, 3]. Interplay between the dynamics of Kalman-filter-like architectures and learning of parameters of neuronal networks has promising aspects for explaining known and puzzling phenomena, such as priming, repetition suppression and categorization [4, 1]. As it is well known, Kalman-filter provides an on-line estimation of the state of the system. On the other hand, optimal control cannot be computed on-line, because it is typically given by a backward recursion (the Ricatti-equations). For on-line parameter estimations without control aspects, see [5]. The aim of this paper is to derive an on-line control method for the Kalman-filter and achieve optimal performance asymptotically. Slight modification of the linearquadratic-Gaussian (LQG) Kalman-filter model is introduced for treating the LQG model as a reinforcement learning (RL) problem. 2. the Kalman filter and the LQG model Consider a linear dynamical system with state xt ∈ R, control ut ∈ R, observation yt ∈ R, noises wt ∈ R and et ∈ R (which are assumed to be Gaussian